WARMStones: Benchmarking Wide-Area Resource Management Schedulers

Researchers have proposed hundreds of algorithms for solutions to the distributed scheduling, or matching, problem. However, few of these algorithms have been implemented, and the primary analysis has either been purely theoretical or with limited simulation. There has been little or no performance analysis of these algorithms under realistic workloads. The result is that it is impossible to compare algorithms and to determine which work well in practice. We propose to build a benchmarking system for matching algorithms with the following properties: (1) the system will use real, existing high-performance computing applications as a basis for the benchmark suite, (2) we will provide a scheduler implementation toolkit, allowing researchers to implement their algorithms in isolation from dependencies on particular scheduling support systems, and (3) we will simulate the execution of the matching algorithm on a variety of heterogeneous distributed system architectures. The result will be that we can assess the strengths and weaknesses of individual scheduling algorithms in a uniform manner, thus allowing comparison of algorithm performance under realistic workloads.